Cooperative augmented reality map interface

ABSTRACT

To reduce this amount of bandwidth needed to share 3D map images between mobile devices, according to some embodiments, a user&#39;s mobile device (i.e., a host device) may identify its origin in a 3D map and a current virtual camera position relative to the origin based on the physical position of the mobile device. The mobile device may send both the origin and the virtual camera position to another mobile device (i.e., a client device) for use in rendering a corresponding image. Separately, the client device may download the 3D map images from a server, e.g., in preparation for a meeting. In this manner, the host device may send the origin to the client device once, as well as send a data stream of the current virtual camera position for use in accessing the corresponding 3D map images at the client device.

CROSS-REFERENCES TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.17/013,344, entitled “Cooperative Augmented Reality Map Interface,”which is a continuation of U.S. application Ser. No. 15/867,351 entitled“Cooperative Augmented Reality Map Interface,” which claims the benefitof U.S. Provisional Application No. 62/566,184, filed Sep. 29, 2017,entitled “Cooperative Augmented Reality Map Interface,” the disclosureof this application is incorporated by reference herein in theirentirety and for all purposes. This disclosure is also related to U.S.Provisional Application No. 62/521,895, filed Jun. 19, 2017, thedisclosure of which is incorporated by reference herein in its entirety.

BACKGROUND

Modern phones can provide location information via a map application.The view of a map is typically a top down view (i.e., an overhead view)of a grid of streets. The top down view may also provide a satelliteimage. When a user is at a particular view of a map, the user may wishto have a three dimensional (3D) view. The 3D view can be provided froma perspective of a virtual camera. Typically, the position of thevirtual camera is specified via a touchscreen interface through gestures(i.e., a touch, a drag, a rotation, etc.).

BRIEF SUMMARY

It may be convenient to view a 3D map on a mobile device (e.g., a phone)where the location of the mobile device changes the view of the 3D mapby moving the perspective of the virtual camera, e.g., via an augmentedreality (AR) interface or a virtual reality (VR) interface. The VRinterface may display the 3D map, while the AR interface may have anoverlay of the 3D map on camera images captures by the phone.

A user of a mobile device may want to send his or her particular view ofthe 3D map from his or her mobile device to another user's mobile devicein order to share the particular view. For example, a city planner maywant to take someone on a virtual guided tour of a 3D map, where theimages are generated using the user interface. Such user-generatedvideos may be transmitted over a network, which may use a large amountof bandwidth. Thus, some embodiments provide systems and methods ofsharing 3D map views at a reduced bandwidth.

In some embodiments, a method is provided. The method comprises a clientdevice retrieving a set of map objects corresponding to a region of amap. In some embodiments, the map may be a three-dimensional (3D) map.The client device may receive an origin position in the region of themap over a network. The client device may receive, over a network, astream of virtual positions of the host device in the map. The virtualmap may be 6D positions. The client device may render and display aseries of images of the map using the stream of virtual positions of thehost device. The client device may use the stream of virtual positionsto determine which map objects to retrieve from memory and display.

In some embodiments, a method for implementing a cooperative discoverymode on a map interface of a mobile device are provided. A selection ofa discovery mode may be received via a user interface of a clientdevice. The discovery mode may indicate, for example, that the clientdevice does not want to follow the virtual positions of a host device,and instead wants to navigate map images independently. One or morefirst images of a physical environment within which the client deviceresides may be captured using a physical camera of the client device. Aninitial position of the physical camera in the physical environment maybe determined using the one or more first images. The initial positionof the physical camera can be defined relative to an origin position inthe physical environment. A set of physical positions of a set of 3Dobjects of a 3D map relative to the initial position may be specifiedbased on an origin position from a host device and a set of map objectsfrom a map server. The set of three-dimensional objects may correspondto the set of map objects. A stream of virtual positions of a clientvirtual camera corresponding to the client device may be generated basedon physical movement of the client device as measured using one or moresecond images of the physical environment. In other words, the physicalmovement of the client device may be determined by consecutive imagestaken by the physical camera of the client device. A series of images ofthe 3D map may be rendered and displayed using the stream of virtualpositions of the client virtual camera. In this case, the physicalmovement of the client device to be closer to the computer may betranslated into a zoomed-in view of the three-dimensional map. Thus,movement of the client device can allow a user to control what parts ofa map (e.g., a three-dimensional map) are displayed on the mobiledevice. This control of what is displayed on the client device may beindependent of what is being displayed on the host device, in someembodiments

In some embodiments, a method of providing a view of a three-dimensional(3D) map on a display of a client device is provided. The methodcomprises performing, by the client device having a physical cameracommunicably coupled with the display: retrieving, from a map server, astep of map objects corresponding to a region of the 3D map; receiving,over a network, an origin position in the region of the 3D map, theorigin position being specified by a host device; receiving, over thenetwork, a first stream of virtual positions of a host virtual cameracorresponding to the host device in the 3D map, wherein the virtualpositions are relative to the origin position and are generated byphysical movement of the host device; rendering and displaying a firstseries of images of the 3D map using the first stream of virtualpositions of the host device; and providing a user interface for a userto select a discovery mode that uses the physical camera. The methodfurther comprises, in response to a selection of the discovery mode viathe user interface: capturing one or more first images of a physicalenvironment within which the client device resides using the physicalcamera; determining an initial position of the physical camera in thephysical environment using the one or more first images; specifying aset of physical positions of a set of 3D objects of the 3D map relativeto the initial position based on the origin position and the set of mapobjects; and generating a second stream of virtual positions of a clientvirtual camera corresponding to the client device based on physicalmovement of the receiving mobile device as measured using one or moresecond images from the physical camera. The method further comprisesrendering and displaying a second series of images of the 3D map usingthe second stream of virtual positions of the client virtual camera.

In some embodiments, a mobile device is provided. The mobile devicecomprises a display, a physical camera communicably coupled with thedisplay, a processor and a memory coupled to the processor, the memorystoring instructions, which when executed by the processor, cause themobile device to perform operations including the steps of the disclosedmethods, for example.

In some embodiments, a computer-program product is provided. Thecomputer-program product is tangibly embodied in a non-transitorymachine-readable storage medium of a host device, including instructionsthat, when executed by one or more processors, cause the one or moreprocessors to perform operations including the steps of the disclosedmethods, for example.

The following detailed description together with the accompanyingdrawings in which the same reference numerals are sometimes used inmultiple figures to designate similar or identical structural elements,provide a better understanding of the nature and advantages of thepresent disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a block diagram depicting a mobile device implementingaugmented reality according to some embodiments of the presentinvention.

FIG. 2A shows a diagram depicting a user using a mobile device toregister and view an augmented reality virtual map as if athree-dimensional map existed in a room according to some embodiments ofthe present invention.

FIG. 2B shows a diagram depicting a user using a movement-controlleduser interface of the augmented reality virtual map to view thethree-dimensional map from a different perspective with respect to FIG.2A according to some embodiments of the invention.

FIG. 2C shows a movement of a virtual camera to view a virtual map ascontrolled by movement of a mobile device according to some embodimentsof the present invention.

FIG. 3 shows a display depicting a zoomed in, three-dimensionalsatellite view from the perspective of a virtual camera in a mapapplication according to some embodiments of the present invention.

FIG. 4 shows a display depicting a zoomed in, shifted three-dimensionalsatellite view from the perspective of a virtual camera in a mapapplication according to some embodiments of the present invention.

FIG. 5A shows a flow chart depicting a method for implementing afollower mode on a map interface of a mobile device according to someembodiments of the present invention.

FIG. 5B shows a block diagram depicting a host device and a clientdevice implementing cooperative map interfaces according to someembodiments of the present invention.

FIG. 6A shows a diagram depicting a client device and a host devicebeing used to view an augmented reality map as if a three-dimensionalmap existed in a room according to some embodiments of the presentinvention.

FIG. 6B shows a screen shot of a map image on a client device and a hostdevice from the positions shown in FIG. 6A in a follower mode accordingto some embodiments of the present invention.

FIG. 7 shows a display depicting movement of a virtual camera of a hostdevice through a three-dimensional satellite view with a client devicefollowing the movement of the host device according to some embodimentsof the present invention.

FIG. 8 shows a display depicting movement of a virtual camera of a hostdevice through a three-dimensional satellite view with a client devicein a discovery mode according to some embodiments of the presentinvention.

FIG. 9 shows a flow chart depicting a method for implementing adiscovery mode on a map interface of a mobile device according to someembodiments of the present invention.

FIG. 10A shows a screen shot of a map image on a host device from theposition shown in FIG. 6A in a discovery mode according to someembodiments of the present invention.

FIG. 10B shows a screen shot of a map image on a client device from theposition shown in FIG. 6B in a discovery mode according to someembodiments of the present invention.

FIG. 11 shows a flow chart depicting a method for combined follower anddiscovery modes on a map interface of a mobile device according to someembodiments of the present invention.

FIG. 12 shows a block diagram depicting a host device and a clientdevice implementing cooperative map interfaces according to someembodiments of the present invention.

FIG. 13 shows a block diagram depicting a server computer, which may bea map server, according to some embodiments of the present invention.

FIG. 14 shows a block diagram depicting a mobile device, which may be ahost device or a client device, according to some embodiments of thepresent invention.

FIG. 15 shows a block diagram of an example device, which may be amobile device, according to some embodiments of the present invention.

DETAILED DESCRIPTION

To reduce the amount of bandwidth needed to share 3D map images betweenmobile devices (as may be generated using a position with orientation ofa mobile device), a user's mobile device (i.e., a host device) canidentify its origin in a 3D map and a current virtual camera positionrelative to the origin based on the physical position of the mobiledevice. The mobile device may send both the origin and the virtualcamera position to another mobile device (i.e., a client device) for usein rendering a corresponding image. Separately, the client device maydownload the 3D map images from a server, e.g., in preparation for ameeting. In this manner, the host device may send the origin to theclient device once, as well as send a data stream of the current virtualcamera position for use in accessing the corresponding 3D map images atthe client device. This mode may be referred to herein as a “followermode”.

In addition, in some embodiments, the client device may have an optionto control its own view of the 3D map images. In such an option, thecurrent virtual camera position of the host device in the 3D map may bedisplayed (e.g., some marker or icon can be displayed, potentially withan orientation indicator). In this manner, the client device may viewwhat its user wants to see, while still being aware of what the user ofthe host device is seeing. In addition, the view of the client devicemay be able to be aligned with the view of the host device based on thedisplayed location of the host device. This mode may be referred toherein as a “discovery mode”.

In some embodiments, the host device and the client device may move inand out of the follower mode and the discovery mode. In the discoverymode, the host device's position may be displayed as a trace over time,showing a path of the host device. Such a path may be considered a “tourguide path”. The client device may leave and re-enter along such a tourguide path at various positions along that path, while still having thefreedom to discover other portions of the 3D map.

Embodiments of the present invention provide a number of advantages. Forexample, new users of map applications may find traditional methods ofmanipulating maps to be difficult. A user may be familiar withone-finger manipulations but may be unfamiliar with more complicatedtwo-finger manipulations. Thus, some embodiments of the inventionprovide an interface for interacting with a displayed map that is easyand intuitive, allowing users to interact with the displayed map bymoving the mobile device. In addition, users may share theirinteractions with the displayed map on the mobile device throughcoordination with another mobile device. This may make it unnecessary toshare potentially multiple different addresses, coordinates, or pointsof interest between users, and allow one user to simply “follow” anotheruser's interactions with the map.

I. Augmented and Virtual Reality

Augmented reality describes a technology in which a live view of thereal world is supplemented with computer-generated data, such as text,graphics, or audio. In other words, the real world as seen by anaugmented reality device is enhanced with additional features. With theuse of augmented reality, the real world may become interactive andinformative. For example, information about an object in a real-worldscene may be overlaid onto the real-world scene to provide the user withmore information about the viewed object.

Virtual reality describes a technology in which a computer-generatedsimulation of an image may be interacted with using real worldmovements, gestures or actions. For example, realistic images may beused to simulate a user's presence in a virtual environment. The usermay be able to interact with the virtual environment, such as by turninghis head to look around the virtual environment, or by extending hishand toward a virtual item to virtually touch or move the item.

In some embodiments, augmented or virtual reality may be implemented ona mobile device. FIG. 1 shows a block diagram depicting a mobile device100 implementing augmented reality according to some embodiments of thepresent invention. In some embodiments, the mobile device 100 may be acommunication device that may provide remote communication capabilitiesto a network. Examples of remote communication capabilities includeusing a mobile phone (wireless) network, wireless data network (e.g.,3G, 4G or similar networks), WiFi, Wi-Max, or any other communicationmedium that may provide access to a network such as the Internet or aprivate network. Examples of mobile devices 100 include mobile phones(e.g., cellular phones), PDAs, tablet computers, net books, laptopcomputers, personal music players, handheld specialized readers,watches, fitness bands, wearables, etc., as well as automobiles withremote communication capabilities. The mobile device 100 may compriseany suitable hardware and software for performing such functions and mayalso include multiple devices or components (e.g., when a device hasremote access to a network by tethering to another device—i.e., usingthe other device as a modem—both devices taken together may beconsidered a single communication device).

The mobile device 100 may include hardware components, such as a camera105, a processor 110, a memory 115, and a display 120. Althoughillustrated and described as being internal to the mobile device 100, itis contemplated that any or all of these hardware components mayalternatively or additionally be implemented external to and/or remotefrom the mobile device 100. To implement augmented reality, the camera105 may be used to capture image data 107 of a real-world view. Forexample, the camera 105 may capture a moveable image of the environmentpositioned in front of the camera 105 at a given moment. The camera 105may transmit the image data 107, as a single image or a video, to theprocessor 110. The camera 105 may be a physical camera.

The processor 110 may process the image data 107 to extract features 113from the image data 107. The processor 110 may analyze the image data107 to determine whether particular objects are present in the imagedata 107. For example, the processor 110 may run a classifier on theimage data 107 to identify images of people in the image data 107 basedon common features of people (e.g., a head, a face, a body, arms, legs,hands, feet, typical movement profiles, etc.). Similarly, the processor110 may run a classifier on the image data 107 to identify otherparticular objects present in the image data 107. These identifiedobjects may be passed as features 113 to the memory 115.

The processor 110 may use the features 113 to retrieve augmented data117 from the memory 115. For example, a feature 113 may be “dog”. Asfacilitated by the processor 110, the memory 115 may be searched fordatabase entries corresponding to “dog”. One or more of the databaseentries may be passed back to the processor 110 as augmented data 117.The augmented data 117 may include any data relevant to the features113, such as text (e.g., a description, a definition, a website address,etc.), a graphic, audio, video, an interactive element, and/or the like.

The processor 110 may receive the augmented data 117 and overlay theaugmented data 117 onto the image data 107. The augmented data 117 mayspecify the features 113 to which the augmented data 117 is relevant.Thus, the processor 110 may locate the features 113 in the image data107 and overlay the augmented data 117 at a particular location. Asexamples, the particular location may be proximate to the relevantfeature 113, be overlapping with the relevant feature 113, be associatedwith the relevant feature 113 (e.g., with an arrow, point, pin,highlight, or other indicator to the feature 113), be in a popup box orwindow, and/or the like.

The image data 107 with the overlaid augmented data 117 may togetherform an augmented image 118 that is transmitted to the display 120. Thedisplay 120 may display the augmented image 118 on the mobile device100. In some embodiments, the display 120 may allow interaction with theaugmented image 118, such as zooming in, zooming out, cropping,selecting a link (e.g., to a website or file), modifying, editing,and/or the like. This interaction may be facilitated by an input element(not shown) that provides input commands to the processor 110, such as atouchscreen element incorporated into the display, a mouse, a trackpad,a trackball, a keyboard, a microphone, and/or the like.

II. Map Applications

A map application may be implemented on a mobile device to assist a userin finding a location. The map application may display a map of theuser's current location or an input location. The user may enter anaddress, drop a pin, and/or search for another location in the mapapplication. The map application may display the location and, in someembodiments, allow manipulation of the view of the map displaying thelocation. For example, the map application may allow the user to zoomin, zoom out, rotate, display labels, hide labels, etc. The mapapplication may further allow the user to perform one or more functionsrelative to the displayed location, such as to calculate and displaydirections from the user's current location to the displayed location,display traffic, change from a two dimensional to a three dimensionalview and vice versa, change from a map view to a satellite view and viceversa, etc.

Map applications may implement graphical user interfaces, such as thosedepicting a two-dimensional map view in a map application. An addressmay be entered into the map application (e.g., “2 Embarcadero Center,San Francisco, Calif.”). The map application may retrieve and displaymap data including the entered address. In some examples, the mapapplication may display the map data in a two-dimensional map view. Themap application may further drop a pin or other indicator at the enteredaddress. The map data may include roads, road names, cities, city names,neighborhoods, landmarks, parks, businesses, public transportation,and/or the like.

In some examples, the graphical user interface may be zoomed in to showa closer view of the entered address. The zoomed in view may providemore detailed map data, such as more road names and more features (e.g.,businesses and landmarks), with respect to what is shown in the originalmap view. To zoom in, a user may use his fingertips to manipulate theoriginal map. For example, the user may place two fingertips on theoriginal map, then spread his fingertips apart in order to zoom in andarrive at a more detailed map. This motion may not be intuitive for someusers.

III. Movement Interface for Interacting with Map

Some embodiments of the present invention provide a virtual realityinterface or augmented reality interface in which the map is the virtualenvironment and the movement of the mobile device is the user's realworld interaction with the virtual environment. For example, embodimentsprovide a movement-controlled user interface that allows for intuitiveinteraction with a displayed map, without requiring fingertipmanipulation on the display. The interaction may be made by physicallymoving (e.g., through translation and/or rotation of) the mobile deviceupon which the map is displayed. For example, the map may be zoomed inwhen the mobile device is moved away from the user (e.g., toward anobject in view), and zoomed out when the mobile device is moved towardthe user (e.g., away from an object in view). Similarly, the map may beshifted left when the mobile device is moved leftward, up when themobile device is rotated upward, and the like. Display of virtual mapbased on movement of mobile device

In some embodiments, an interface may be provided in which a virtual mapimage may be displayed on a mobile device, and in some embodiments,overlaid onto a real-world image. FIG. 2A shows a diagram depicting auser 202 using a mobile device to view a virtual map according to someembodiments of the present invention. Specifically, the user 202 may usea physical camera of the mobile device at initial position 205 tocapture an image of the desk 230. The mobile device may identify desk230 as a suitable surface (e.g., a horizontal surface or flat surface)on which to overlay a map image including three-dimensional map objects(e.g., building 215, tree 220, and bank 225). Thus, the mobile devicemay display the three-dimensional map objects onto the desk 230 as seenon the display of the mobile device. Building 215, tree 220, and bank225 may appear to be positioned on the desk 230 such that the base ofthe objects appear to be positioned on the desk 230 with the objectsprotruding from the desk 230.

The dashed lines in FIG. 2A indicate that these map objects are onlyseen by a user when viewing a display of the mobile device, and thatthese map objects are not physically present on the desk 230. In FIG.2A, the building 215 may appear to be closest to the user 202, followedby the tree 220 and the bank 225, e.g., when the user 202 is viewing thedisplay of the mobile device as a camera of the mobile device is pointedat the desk 230. Although shown and described herein with respect to adesk 230, it is contemplated that building 215, tree 220, and bank 225may be rendered onto any arbitrary horizontal plane. Further, it iscontemplated that the rendering of building 215, tree 220, and bank 225may not be limited to a set region (e.g., the bounds of desk 230), butmay rather render to the user 202 as if the image goes to the horizon.

The physical camera may have a viewpoint represented by a registrationvector 207 at which the tree 220 is at a default origin position. Thedefault origin position may be, for example, the central point in thephysical camera's initial field of view. The origin position may reflectthe initial view of the physical camera, with the location of the mapobjects defined with respect to that origin position. For example, thebuilding 215 may be defined at a position three inches to the left andone inch ahead of the origin position, the tree 220 may be defined atthe origin position, and the bank 225 may be defined at a location twoinches to the right and one inch behind the origin position. Thesepositions may be scaled down or translated from real world distances.For example, the building 215 may be 100 feet from the origin position,but be scaled down to three inches for purposes of position on desk 230.These locations with respect to the origin position may be used toregister the positions of the map objects relative to any movement ofthe mobile device at initial position 205.

The mobile device may then be moved around the desk 230, still pointedat the desk 230, and the map image may be continuously rendered toreflect the changes in position and orientation, such that the map imageappears to be an interactive three-dimensional model on the desk 230.Such movement of the mobile device can control a virtual position of avirtual camera that is used to render the images of the virtualthree-dimensional model. The three-dimensional model may be moved,repositioned, zoomed, and otherwise interacted with by a user viamovement of the mobile device implementing the augmented realityinterface.

FIG. 2B shows a diagram depicting the user 202 using the mobile deviceto view an augmented reality virtual map from a current position 210with respect to FIG. 2A according to some embodiments of the invention.In FIG. 2B, the user 202 has moved the mobile device from initialposition 205 to current position 210. This change in position fromposition 205 to position 210 may be represented by a movement vector 209that shows the movement of the mobile device from the left side of thedesk 230 to the right side of the desk 230. The movement vector 209 maydefine movement of the mobile device relative to its initial position205. The current position 210 of the mobile device may be defined by itsinitial position 205 relative to the origin position and the movementvector 209. In some embodiments, the movement vector 209 may be scaledto the map image to reflect the movement with respect to thethree-dimensional map objects (e.g., building 215, tree 220, and bank225). In other words, the movement vector 209 may be scaled to adifferent amount of movement of the map objects. For example, every inchof movement of the physical camera may be scaled to five feet ofmovement of the virtual camera used to view the map objects. Aftermoving to the current position to the right of the desk 230, the bank225 may appear closest to the user 202, followed by the tree 220 and thebuilding 215.

FIG. 2C shows movement of a virtual camera to view an augmented realityvirtual map as controlled by movement of a mobile device according toembodiments of the present invention. As stated above, the mobile devicemay have a physical camera that may capture images of the real world.The mobile device may be associated with a virtual camera at an initialposition 216 in the virtual map environment, the perspective of which isused to display the map objects (e.g., building 215, tree 220, and bank225). The origin position and registration vector 207 defined by thephysical camera may correspond to an origin position and registrationvector 211 defined by the virtual camera. The origin position defined bythe virtual camera may be a position in the virtual map around which thepositions of map objects are registered. The registration vector 211 maydefine the initial position 216 and perspective of the virtual camerawith respect to the origin position.

When the mobile device is moved and rotated around the desk 230, asshown in FIG. 2B, the map image may be continuously rendered to reflectthe changes in position and orientation. For example, the mobile devicemay be moved from an initial position 205 and orientation to a currentposition 210 and orientation. Correspondingly, the virtual camera may bemoved from initial position 216 and orientation to current position 217and orientation. Orientation vector 213 may illustrate the rotation ofthe virtual camera in association with a rotation of the mobile devicefrom an initial position 205 to a current position 210, which may bedefined as a 6-dimensional vector of 3 translation coordinates and 3rotation angles. The map objects (e.g., building 215, tree 220, and bank225) may then be rendered from the perspective of the virtual camera atcurrent position 217, at its current orientation. Thus, the virtualcamera at the initial position 216 may view an image of the backside ofbuilding 215, tree 220, and bank 225 from nearest to farthest (e.g.,corresponding to position of the mobile device 205 in FIG. 2A), whilethe virtual camera at the current position 217 may view an image of thefront side of bank 225, tree 220, and building 215 from nearest tofarthest (e.g., corresponding to position of the mobile device 205 inFIG. 2B). A height of the mobile device or a distance between the mobiledevice and the desk 230 can also control the height of the virtualcamera.

FIGS. 2A-C describe the rendering of the virtual building 215, tree 220,and bank 225 onto the camera-captured image of the real-world desk 230.However, it is contemplated that, in some embodiments, the imagescaptured by the camera of the mobile device 205 are not displayed on themobile device 205. In other words, the virtual map including the virtualbuilding 215, tree 220, and bank 225 may not be overlaid onto a realworld image (e.g., onto real world desk 230), and may be displayedwithout the real world image.

A. Map Interface

The initial three-dimensional view of a map from the perspective of thevirtual camera may be displayed in response to a location being detectedor an address being entered. FIG. 3 shows a graphical user interfacedepicting a zoomed out, three-dimensional satellite map view map image300 from the perspective of a virtual camera (e.g., virtual camera atinitial position 216) in a map application according to some embodimentsof the present invention. As shown in FIG. 3, an address has beenentered into the map application (e.g., “2 Embarcadero Center, SanFrancisco, Calif.”). The map application may retrieve and display mapdata including the entered address.

In this example, the map application may display the map data in athree-dimensional satellite (e.g., overhead) view. The map applicationmay further drop a pin or other indicator at the entered address. Themap data may include roads, road names, cities, city names,neighborhoods, landmarks, parks, businesses, public transportation,and/or the like (e.g., map objects such as building 215, tree 220, andbank 225 of FIGS. 2A-C). In some embodiments, the three-dimensionalsatellite view of FIG. 3 may be displayed in response to a userselection of a three-dimensional satellite view from the graphical userinterface.

The map image 300 can correspond to the initial position of a virtualcamera, e.g., initial position 216 of FIG. 2C. The physical camera canhave a corresponding initial position from which movement is compared.Movement from this initial position of the mobile device as captured bythe physical camera and/or sensors may result in corresponding movementin the displayed map.

FIG. 4 shows a graphical user interface depicting a zoomed in,three-dimensional satellite map image 400 from the perspective of avirtual camera (e.g., the virtual camera at position 217 of FIG. 2C) ina map application according to some embodiments of the presentinvention. The perspective of the virtual camera of FIG. 4 has beenshifted with respect to FIG. 3. Specifically, the virtual camera of FIG.4 has been zoomed in and shifted to the left. As an example, suchmovement can correspond to a user taking a step to the left and a stepforward. In this manner, the user can control which map objects aredisplayed on a screen of the mobile device, as well as how they aredisplayed (e.g., what zoom level or what virtual camera angle is used)by moving the mobile device.

IV. Cooperative Map Interface

In some embodiments, a user's mobile device (i.e., a host device) mayidentify its origin in a 3D map and a current virtual camera positionrelative to the origin based on the physical position of the mobiledevice. The mobile device may send both the origin and the virtualcamera position to a second mobile device (i.e., a client device) foruse in rendering a corresponding image. Separately, the client devicemay download the 3D map images from a server, e.g., in preparation for ameeting. In this manner, the host device may send the origin to theclient device once, as well as send a data stream of the current virtualcamera position for use in accessing the corresponding 3D map images atthe client device. In some implementations, the mobile device can sendthe origin and virtual camera position to a server, and the server cansend the origin and virtual camera position to the second mobile device.The second mobile device can receive a stream of the virtual camerapositions.

A. Follower Mode

A user's movement interactions with the map interface (e.g., movement ofthe mobile device causing movement of the displayed map) may be sharedamongst one or more other mobile devices. For example, a first user cannavigate or interact with a map image and share the experience with asecond user. This allows the second user to see the same map objects asthe first user and have the same experience. In some implementations, asending mobile device (i.e., a host device) may transmit an originposition in a region of a map to one or more receiving mobile devices(i.e., a client device). This origin position may be sent initially to aclient device or after the client device downloads a set of map imagescorresponding to an initial geographic indicator. The origin positionmay correspond to, for example, the current position of the host devicewith respect to the virtual world, an address entered by the hostdevice, coordinates entered by the host device, a point of interestentered by the host device, a pin dropped by the host device, etc. Inanother example, the origin position may be a default position based onthe position of the host device as captured by a physical camera withrespect to a map image. For example, as shown in FIG. 3, the originposition 302 may correspond to “2 Embarcadero Center”.

In response to receiving the origin position 302, the client device maydownload a set of map objects corresponding to the region of the mapincluding the origin position. In some embodiments, the client devicemay include a rendering engine that uses the map objects and the originposition to generate the map images. In some embodiments, the clientdevice may download these map objects from a map server or other servercomputer, thereby reducing bandwidth needed between the host device andclient device. In some embodiments, however, the client device maydownload some or all of these map objects from the host device. Forexample, the client device may download the map image 300 shown in FIG.3, as well as other map images surrounding the origin position 302. Forexample, the client device may download map images within a certainradius of the origin position 302, e.g., map images and objects within amile of the origin position 302. Thus, the client device may have themost likely map objects needed for display available locally.

The host device may transmit a stream of virtual positions of the hostdevice in the map corresponding to physical movements of the hostdevice. For example, as shown in FIGS. 3 and 4 and described above, thehost device may physically move forward and to the left, causingmanipulation of the displayed map 400 as shown in FIG. 4. Specifically,the map may zoom in and pan left with respect to the origin position302, causing a stream of virtual positions indicated by trace 304. Thecurrent virtual position of the host device may be indicated by anindicator 306.

In response to receiving the stream of virtual positions of the hostdevice, the client device may render and display a series of map imagesusing the stream of virtual positions. For example, the client devicemay render and display the zooming in and movement to the left of themap image shown in FIG. 3 along the trace 304 as a series of map images.The movement may stop at the current virtual position of the host deviceas indicated by the indicator 306. Thus, the map image 400 shown in FIG.4 may be displayed on both the host device and the client device.

FIG. 5A shows a flow chart 500 depicting a method for implementing afollower mode on a map interface of a mobile device according to someembodiments of the present invention. At step 505, a client device mayretrieve a set of map objects corresponding to a region of a map. Insome embodiments, the map may be a three-dimensional (3D) map. In someembodiments, the client device may retrieve the set of map objects froma map server or other remote server computer, freeing up bandwidthbetween the client device and a host device. In some embodiments, theclient device may retrieve the set of map objects from a host device.Although referred to herein as “a” host device and “a” client device, itis contemplated that one or more host device and one or more clientdevice may be used in similar embodiments.

At step 510, the client device may receive an origin position in theregion of the map over a network. The network may be any type of localor remote communication network, such as, for example, WiFi, Bluetooth,near field communication (NFC), a cellular network, a virtual privatenetwork (VPN), the Internet, or any other suitable network. The originposition may be specified by the host device. For example, the hostdevice may select its current position, may enter an address, may entercoordinates, may enter a point of interest, may select a location from amap image, e.g., by dropping a pin on the map image, etc. In someembodiments, step 510 may be performed prior to step 505, such that theclient device downloads the set of map objects most relevant to and/orproximate to the origin position.

At step 515, the client device may receive, over a network, a stream ofvirtual positions of the host device in the map. The network may be thesame or a different network. For example, the client device may receivethe origin position of the host device via Bluetooth, but receive thestream of virtual positions over WiFi due to decreased or diminishedBluetooth signal strength. The stream of virtual positions in the mapmay correspond to physical movements of the host device in the realworld. For example, physically moving the host device to the right maygenerate a stream of virtual positions in the map moving down the streetto the right of the origin position. The stream of virtual positions maytogether make up a “trace” indicating the virtual movement of the hostdevice in the map.

At step 520, the client device may render and display a series of imagesof the map using the stream of virtual positions of the host device. Theclient device may use the stream of virtual positions to determine whichmap objects to retrieve from memory and display. For example, if thestream of virtual positions in the map indicates that the center of themap should be moved from the origin position down the street to theright of the origin position by 500 feet, map objects along that tracemay be retrieved, rendered, and displayed on the client device. Thus,the client device may “follow” the movement of a host device within amap based on physical movements of the host device.

FIG. 5B shows a block diagram depicting a host device 535 and a clientdevice 540 implementing cooperative map interfaces according to someembodiments of the present invention. As shown in FIG. 5B, the hostdevice 535 may transmit a request 526 to a map server 525 for mapimages. The request may include a geographic indicator of the desiredmap images, such as, for example, an origin position, an address, a setof coordinates, a pin, etc. In response to the request 526, the hostdevice 535 may download map images 527 from the map server 525. The mapserver 525 may be any server computer and/or database storing mapimages. In some embodiments, the map images 527 received from the mapserver 525 may be selected based on, for example, proximity to theorigin position 536 entered by the host device 535. The origin position536 may be selected, for example, based on the current position of thehost device 535, an address or coordinates entered by the host device535, a pin dropped on a map displayed on the host device 535, etc. Themap images 527 may be displayed on the host device, and in someembodiments, centered on the origin position 536.

The host device 535 may transmit the origin position 536 to the clientdevice 540. Based on the origin position 536 and/or a geographicindicator, the client device 540 may download the map images 539 fromthe map server 525. Because the map images 539 may be downloaded fromthe map server 525 instead of the host device 535, bandwidth may bepreserved between the host device 535 and the client device 540. The mapimages 539 may be displayed on the client device, and in someembodiments, centered on the origin position 536.

The host device 535 may be physically moved and/or manipulated in orderto cause movement and/or manipulation of the displayed map images. Inother words, the positions 537 of a virtual camera of the host devicevirtually pointed at the map images may be moved and/or manipulatedbased on the physical movement of the host device 535. These virtualcamera positions 537 may be transmitted to the client device 540. Theclient device 540 may then transmit a request 538 to the map server 525for map images 539 in proximity of the geographic indicator. The clientdevice 540 may then render and display the same movements of the mapimages using the virtual camera positions 537 of the host device 535.

In some embodiments, the client device 540 may also enter a “discoverymode” in which the client device 540 may control the map images 539displayed on the client device 540, independent of the map images 527displayed on the host device 535. In these embodiments, the physicalcamera of the client device 540 may be used to capture initial andsubsequent images of the real world (or other sensors may be used tocapture movement of the client device, such as an accelerometer).Corresponding movement of the map images 539 may be made based on thephysical movement of the client device 540. Thus, in “discovery mode”,the map images 539 displayed on the client device 540 may be differentthan the map images 527 displayed on the host device 535, as the clientdevice 540 is in control of its own displayed map images 539.

FIG. 6A shows a diagram depicting a client device 607 and a host device605 being used to view an augmented reality map as if athree-dimensional map existed in a room according to some embodiments ofthe present invention. In some embodiments, an interface may be providedin which a virtual map image may be displayed on a mobile device, and insome embodiments, overlaid onto a real-world image by the host deviceand/or the client device. Specifically, a user may use a physical cameraof the host device at initial position 205 to capture an image of thedesk 630. The host device may identify desk 630 as a suitable surface(e.g., a horizontal surface or flat surface) on which to overlay a mapimage including three-dimensional map objects (e.g., building 615, tree620, and bank 625). Thus, the host device may display thethree-dimensional map objects onto the desk 630 as seen on the displayof the host device. Building 615, tree 620, and bank 625 may appear tobe positioned on the desk 630 such that the base of the objects appearto be positioned on the desk 630 with the objects protruding from thedesk 230.

The dashed lines in FIG. 6A indicate that these map objects are onlyseen by a user when viewing a display of the host device, and that thesemap objects are not physically present on the desk 630. Although shownand described herein with respect to a desk 630, it is contemplated thatbuilding 615, tree 620, and bank 625 may be rendered onto any arbitraryhorizontal plane. Further, it is contemplated that the rendering ofbuilding 615, tree 620, and bank 625 may not be limited to a set region(e.g., the bounds of desk 630), but may rather render to the user as ifthe image goes to the horizon.

The host device may then be moved around the desk 630, still pointed atthe desk 630, and the map image may be continuously rendered to reflectthe changes in position and orientation, such that the map image appearsto be an interactive three-dimensional model on the desk 630. Suchmovement of the mobile device can control a virtual position of avirtual camera that is used to render the images of the virtualthree-dimensional model. The three-dimensional model may be moved,repositioned, zoomed, and otherwise interacted with by a user viamovement of the mobile device implementing the augmented realityinterface. The three-dimensional model may then be displayed as seen bythe host device onto the client device at position 607. Although shownas being in the same room, it is contemplated that the client device andthe host device may be in the same or separate rooms or may render themap image on the same or different surfaces.

FIG. 6B shows a screen shot 600 of a map image on a client device and ahost device from the positions shown in FIG. 6A in a follower modeaccording to some embodiments of the present invention. Specifically,host device at position 605 may view the map image as shown in FIG. 6B.The same map image may be displayed on the client device at position 607when the client device is in a follower mode at position 607.

B. Discovery Mode

In some embodiments, the client device may also have an option tocontrol its own view of the map images. For example, the user of thehost device and the user of the client device can explore the map imageindependently. In such an option, the client device can display an icon,a marker, or other user interface object that indicates the currentvirtual camera position of the host device in the map. In someimplementations, the icon corresponding to the host device can includean orientation indicator (e.g., a cone) that corresponds to the first ofview of the virtual camera. In this manner, the client device may viewwhat its user wants to see, while still being aware of what the user ofthe host device is seeing. In addition, the view of the client devicemay be able to be aligned with the view of the host device based on thedisplayed location of the host device. This mode may be referred to as a“discovery mode”.

FIG. 7 shows a display depicting movement of a virtual camera of a hostdevice through a three-dimensional satellite view map image 700 with aclient device following the movement of the host device according tosome embodiments of the present invention. As described above, the hostdevice may transmit an origin position 710 to the client device. Inresponse to receiving the origin position 710, the client device maydownload a set of map objects corresponding to the region of the mapincluding the origin position or based on other geographical data, suchas a geographical indicator such as an address or pin.

The host device may transmit a stream of virtual positions of the hostdevice in the map corresponding to physical movements of the hostdevice. For example, as shown in FIG. 7, the host device may physicallymove right and forward, causing manipulation of the displayed map.Specifically, the map may shift right and forward with respect to theorigin position 710, causing a stream of virtual positions indicated bytrace 720. The current virtual position of the host device may beindicated by a host indicator 722. In some embodiments, the trace 720may be created when the host device has finished moving, and the clientdevice has retrieved the trace 720 at a later time. In some embodiments,the trace 720 may be created in real time, and the client device may seethe trace 720 as it is being created.

In response to receiving the stream of virtual positions of the hostdevice, the client device may render and display a series of map imagesusing the stream of virtual positions. For example, the client devicemay render and display the movement right and forward along the trace720 as a series of map images. The movement may stop at the currentvirtual position of the host device as indicated by the client indicator724. Thus, the map image 700 shown in FIG. 7 may be displayed on boththe host device and the client device. As shown in FIG. 7, although theclient device may be able to freely move about the map image 700, theclient device may continue to track the host device along the trace 720.For example, the host device may be acting as a tour guide and theclient device may be following the tour of the host device.

In some embodiments, the client device may deviate from the trace 720 ofthe host device to view other map regions not followed by the hostdevice. FIG. 8 shows a display depicting movement of a virtual camera ofa host device through a three-dimensional satellite view map image 800with a client device in a discovery mode according to some embodimentsof the present invention. As described above, the host device maytransmit an origin position 710 to a client device. In response toreceiving the origin position 710, the client device may download a setof map objects corresponding to the region of the map including theorigin position.

The host device may transmit a stream of virtual positions of the hostdevice in the map corresponding to physical movements of the hostdevice. For example, as shown in FIG. 8, the host device may physicallymove right and forward, causing manipulation of the displayed map.Specifically, the map may shift right and forward with respect to theorigin position 710, causing a stream of virtual positions indicated bytrace 720. The current virtual position of the host device may beindicated by a host indicator 822.

In response to receiving the stream of virtual positions of the hostdevice, the client device may render and display a series of map imagesusing the stream of virtual positions. For example, the client devicemay render and display the movement right and forward along the trace720 as a series of map images. However, in this embodiment, the clientdevice may also render and display a series of map images using virtualpositions of the client device. For example, the client device may beginto follow the trace 720, then activate a sensor of the client device tobegin to capture physical movements of the client device. As shown inFIG. 8, for example, the client device may be physically moved to theright and downward in the real world, causing shifting of the map image800 to the right and downward and away from the trace 720, as indicatedby the client indicator 824. However, it is contemplated that the clientdevice may deactivate the sensor and/or deselect a discovery mode inorder to return the client device to the trace 720 at any time.

FIG. 9 shows a flow chart 900 depicting a method for implementing acooperative discovery mode on a map interface of a mobile deviceaccording to some embodiments of the present invention. According to themethod, a host device may begin at an origin position and move along apath of virtual positions in order to create a trace of the virtualpath. A client device may receive this origin position and these virtualpositions and follow the host device in a “follower mode”, as describedfurther herein.

At step 905, a selection of a discovery mode may be received via a userinterface of a client device. The selection of the discovery mode mayoccur, for example, from a user input selection after map objects havebeen downloaded and an origin position has been obtained, as describedfurther herein. The discovery mode may indicate, for example, that theclient device does not want to follow the virtual positions of a hostdevice, and instead wants to navigate map images independently. Theselection of the discovery mode may be made, for example, by selectingan icon on a map image, by activating or selecting a camera on theclient device, etc.

At step 910, one or more first images of a physical environment withinwhich the client device resides may be captured using a physical cameraof the client device. For example, the physical camera may capture anenvironment positioned in front of and around the physical camera.Further details regarding the capturing and interpreting of a physicalenvironment around the physical camera are discussed further herein.

At step 915, an initial position of the physical camera in the physicalenvironment may be determined using the one or more first images. Theinitial position of the physical camera can be defined relative to anorigin position in the physical environment. In some embodiments, theorigin position in the physical environment may correspond to theinitial position of the physical camera when the selection of thediscovery mode is made at step 905. Thus, the origin position in thephysical environment can be taken as the initial position of the mobiledevice 205 in FIG. 2A. As another example, the origin position in thephysical environment can be the center of the map objects (e.g., tree220). As still another example, the origin position in the physicalenvironment may be as defined by the host device.

At step 920, a set of physical positions of a set of 3D objects of a 3Dmap relative to the initial position may be specified based on an originposition from a host device and a set of map objects from a map server.The set of three-dimensional objects may correspond to the set of mapobjects. In some embodiments, the set of physical positions of the setof three-dimensional objects may be specified at default positions fromthe mobile device. The three-dimensional objects may include, forexample, building 215, tree 220, and/or bank 225 of FIGS. 2A-2C. Thedetermination of the physical positions of the 3D objects relative tothe initial position of the physical camera can be defined with respectto the origin position of the host device, which may or may not be theinitial position.

At step 925, a stream of virtual positions of a client virtual cameracorresponding to the client device may be generated based on physicalmovement of the client device as measured using one or more secondimages of the physical environment. In other words, the physicalmovement of the client device may be determined by consecutive imagestaken by the physical camera of the client device. For example, a firstimage may be captured showing a room with a desk, a computer, and abookshelf, where certain map objects (e.g., buildings, street names,etc.) can be displayed on a screen of the client device as if they wereat about the location of the computer. The map objects can be displayedalone, or parts or all of a camera image of the physical world can alsobe displayed. A second, subsequent image may be captured showing onlythe computer (e.g., encompassing more pixels of the image), indicatingthat the user has walked toward the computer, and thus moved toward themap objects. The physical movement of the physical camera of the clientdevice toward the computer may be translated into a virtual movement ofthe client virtual camera in the map application.

At step 930, a series of images of the 3D map may be rendered anddisplayed using the stream of virtual positions of the client virtualcamera. In this case, the physical movement of the client device to becloser to the computer may be translated into a zoomed-in view of thethree-dimensional map. Thus, movement of the client device can allow auser to control what parts of a map (e.g., a three-dimensional map) aredisplayed on the mobile device. This control of what is displayed on theclient device may be independent of what is being displayed on the hostdevice, in some embodiments.

FIG. 10A shows a screen shot 1000A of a map image on a host device fromthe position 605 shown in FIG. 6A in a discovery mode according to someembodiments of the present invention. Specifically, host device atposition 605 may view the map image as shown in FIG. 10A. As shown inFIG. 10A, the host device sees the building 1015 closest to the hostdevice, followed by the tree 1020 and the bank 1025. In someembodiments, the virtual position of the host device in the screen shot1000A may be displayed as an indicator 1005A in the map image.

FIG. 10B shows a screen shot 1000B of a map image on a client devicefrom the position 607 shown in FIG. 6B in a discovery mode according tosome embodiments of the present invention. Because the client device isin a discovery mode, the screen shot 1000B shows a map image from adifferent viewpoint that is independent from the viewpoint of the hostdevice, as reflected in FIG. 10A. As shown in FIG. 10B, the clientdevice sees the bank 1025 closest to the client device, followed by thetree 1020 and the building 1015. In some embodiments, the virtualposition of the host device in the screen shot 1000B may be displayed asan indicator 1005B in the map image. In order for the client device tosee the same map image from the same virtual position as the hostdevice, user input corresponding to a selection of the indicator 1005Bmay be received. By selecting the indicator 1005B on the client device,the displayed map image may move to correspond to screen shot 1000A,such that the view on the client device is the same as the view on thehost device. Thus, the client device may toggle between a discovery modeand a follower mode.

C. Combined Mode

In some embodiments, the client device may toggle between a “followermode” and a “discovery mode”. FIG. 11 shows a flow chart 1100 depictinga method for combined cooperative follower and discovery modes on a mapinterface of a mobile device according to some embodiments of thepresent invention.

At step 1105, a client device may retrieve a set of map objectscorresponding to a region of a map. In some embodiments, the map may bea three-dimensional (3D) map. In some embodiments, the client device mayretrieve the set of map objects from a map server or other remote servercomputer, freeing up bandwidth between the client device and a hostdevice. In some embodiments, the client device may retrieve the set ofmap objects from a host device.

Although referred to herein as “a” host device and “a” client device, itis contemplated that one or more host device and one or more clientdevice may be used in similar embodiments.

At step 1110, the client device may receive an origin position in theregion of the 3D map over a network. The network may be any type oflocal or remote communication network, such as, for example, WiFi,Bluetooth, near field communication (NFC), a cellular network, a virtualprivate network (VPN), or any other suitable network. The originposition may be specified by the host device. For example, the hostdevice may select its current position, may enter an address, may entercoordinates, may enter a point of interest, may select a location from amap image, e.g., by dropping a pin on the map image, etc. In someembodiments, step 1110 may be performed prior to step 1105, such thatthe client device downloads the set of map objects most relevant toand/or proximate to the origin position.

At step 1115, the client device may receive, over a network, a stream ofvirtual positions of the host device in the map. The network may be thesame or a different network. For example, the client device may receivethe origin position of the host device via Bluetooth but receive thestream of virtual positions over Wi-Fi due to decreased or diminishedBluetooth signal strength. The stream of virtual positions in the mapmay correspond to physical movements of the host device in the realworld. For example, physically moving the host device to the right maygenerate a stream of virtual positions in the map moving down the streetto the right of the origin position. The stream of virtual positions maytogether make up a “trace” indicating the virtual movement of the hostdevice in the map.

At step 1120, the client device may render and display a series ofimages of the map using the stream of virtual positions of the hostdevice. The client device may use the stream of virtual positions todetermine which map objects to retrieve from memory and display. Forexample, if the stream of virtual positions in the map indicate that thecenter of the map should be moved from the origin position down thestreet to the right of the origin position by 500 feet, map objectsalong that trace may be retrieved, rendered, and displayed on the clientdevice. Thus, the client device may “follow” the movement of a hostdevice within a map based on physical movements of the host device.

At step 1125, a selection of a discovery mode may be received via a userinterface of the client device. The discovery mode may indicate, forexample, that the client device no longer wants to follow the virtualpositions of the host device, and instead wants to navigate map imagesindependently. The selection of the discovery mode may be made, forexample, by selecting an icon on a map image, by activating or selectinga camera on the client device, etc.

At step 1130, one or more first images of a physical environment withinwhich the client device resides may be captured using a physical cameraof the client device. For example, the physical camera may capture anenvironment positioned in front of and around the physical camera.

At step 1135, an initial position of the physical camera in the physicalenvironment may be determined using the one or more first images. Theinitial position of the physical camera can be defined relative to anorigin position in the physical environment. In some embodiments, theorigin position in the physical environment may correspond to theinitial position of the physical camera when the selection of thediscovery mode is made at step 1125. Thus, the origin position in thephysical environment can be taken as the initial position of the mobiledevice 205 in FIG. 2A. As another example, the origin position in thephysical environment can be the center of the map objects (e.g., tree220). As still another example, the origin position in the physicalenvironment may be as defined by the host device.

At step 1140, a set of physical positions of a set of 3D objects of a 3Dmap relative to the initial position may be specified based on an originposition from a host device and a set of map objects from a map server.The set of three-dimensional objects may correspond to the set of mapobjects. In some embodiments, the set of physical positions of the setof three-dimensional objects may be specified at default positions fromthe mobile device. The three-dimensional objects may include, forexample, building 215, tree 220, and/or bank 225 of FIGS. 2A-2C. Thedetermination of the physical positions of the 3D objects relative tothe initial position of the physical camera can be defined with respectto the origin position of the host device, which may or may not be theinitial position.

At step 1145, a stream of virtual positions of a client virtual cameracorresponding to the client device may be generated based on physicalmovement of the client device as measured using one or more secondimages of the physical environment. In other words, the physicalmovement of the client device may be determined by consecutive imagestaken by the physical camera of the client device. For example, a firstimage may be captured showing a room with a desk, a computer, and abookshelf, where certain map objects (e.g., buildings, street names,etc.) can be displayed on a screen of the client device as if they wereat about the location of the computer. The map objects can be displayedalone, or parts or all of a camera image of the physical world can alsobe displayed. A second, subsequent image may be captured showing onlythe computer (e.g., encompassing more pixels of the image), indicatingthat the user has walked toward the computer, and thus moved toward themap objects. The physical movement of the physical camera of the clientdevice toward the computer may be translated into a virtual movement ofthe client virtual camera in the map application.

At step 1150, a series of images of the 3D map may be rendered anddisplayed using the stream of virtual positions of the client virtualcamera. In this case, the physical movement of the client device to becloser to the computer may be translated into a zoomed-in view of thethree-dimensional map. Thus, movement of the client device can allow auser to control what parts of a map (e.g., a three-dimensional map) aredisplayed on the mobile device. This control of what is displayed on theclient device may be independent of what is being displayed on the hostdevice, in some embodiments, such that the client device can “discover”the map without cooperating with the host device.

V. Systems

Various systems may be used to implement the methods described herein.FIG. 12 shows a block diagram depicting a host device 1205 and a clientdevice 1210 implementing cooperative map interfaces according to someembodiments of the present invention. As shown in FIG. 12, the hostdevice 1205 may download map images 1217 from a map server 1215. The mapserver 1215 may be any server computer and/or database storing mapimages. In some embodiments, the map images 1217 received from the mapserver 1215 may be selected based on, for example, proximity to anorigin position 1207 entered by the host device 1205. The originposition 1207 may be selected, for example, based on the currentposition of the host device 1205, an address or coordinates entered bythe host device 1205, a pin dropped on a map displayed on the hostdevice 1205, etc. The map images 1217 may be displayed on the hostdevice, and in some embodiments, centered on the origin position 1207.

The host device 1205 may transmit the origin position 1207 to the clientdevice 1210. Based on the origin position 1207 (and as described abovewith respect to the host device 1205), the client device 1210 may alsodownload the map images 1218 from the map server 1215.

Because the map images 1218 may be downloaded from the map server 1215instead of the host device 1205, bandwidth may be preserved between thehost device 1205 and the client device 1210. The map images 1218 may bedisplayed on the client device, and in some embodiments, centered on theorigin position 1207.

The host device 1205 may be physically moved and/or manipulated in orderto cause movement and/or manipulation of the displayed map images. Inother words, the positions 1208 of a virtual camera of the host devicevirtually pointed at the map images may be moved and/or manipulatedbased on the physical movement of the host device 1205. These virtualcamera positions 1208 may be transmitted to the client device 1210. Theclient device 1210 may then render and display the same movements of themap images using the virtual camera positions 1208 of the host device1205.

In some embodiments, the client device 1210 may also enter a “discoverymode” in which the client device 1210 may control the map images 1218displayed on the client device 1210, independent of the map images 1217displayed on the host device 1205. In these embodiments, the physicalcamera of the client device 1210 may be used to capture initial andsubsequent images of the real world (or other sensors may be used tocapture movement of the client device, such as an accelerometer).Corresponding movement of the map images 1218 may be made based on thephysical movement of the client device 1210. Thus, in “discovery mode”,the map images 1218 displayed on the client device 1210 may be differentthan the map images 1217 displayed on the host device 1205, as theclient device 1210 is in control of its own displayed map images 1218.

FIG. 13 shows a block diagram depicting a server computer, which may bea map server 1215, according to some embodiments of the presentinvention. The map server 1215 may be in operative communication withone or more mobile devices (e.g., host device 1205, client device 1210,etc.), such as over one or more networks. The network may be any type oflocal or remote communication network, such as, for example, WiFi,Bluetooth, near field communication (NFC), a cellular network, a virtualprivate network (VPN), the Internet, or any other suitable network.

The map server 1215 may include a memory 1316, a processor 1320, and acommunication subsystem 1325. The communication subsystem 1325 mayenable the map server 1215 to communicate with mobile devices, such ashost devices and client devices. The memory may include a map datadatastore 1317. The map data datastore 1317 may store map data(including map images, map objects, etc.) for a plurality of locations.For example, the map data datastore 1317 may store road maps for anumber of regions (e.g., states, countries, etc.).

The map data may include features, such as roads, buildings, businesses,parks, landmarks, houses, trails, and the like. The map data may includemaps in any number of formats (e.g., two dimensional, three-dimensional,map view, satellite view, top down view, eye level view, etc.). The mapdata may be stored along with its associated location(s) in any suitablemanner. For example, the map data may be stored in association withaddresses, coordinates, names, and/or any other indicator.

The map server 1215 may be in communication with a mobile device. FIG.14 shows a block diagram depicting a mobile device 1400, which may be ahost device or a client device as described herein, according to someembodiments of the present invention. The mobile device 1400 may includea memory 1432 and device hardware 1460. The device hardware 1460 mayinclude a processor 1462, a user interface 1464, a global positioningsystem (GPS) 1466, a display 1468, a physical camera 1470, sensors 1472,and a communication subsystem 1474. In some embodiments, the display1468 forms part of the user interface 1464. The user interface 1464 mayfurther include input elements, such as a keyboard, a trackball, atrackpad, a mouse, a microphone, etc. The communication subsystem 1474may include hardware components configured to allow the mobile device1430 to communicate with the map server 1215 and with other mobiledevices, such as over a network.

The memory 1432 may include an operating system (OS) 1450. The operatingsystem 1450 may provide an interface for a user to use the mobile device1400, e.g., to operate device hardware 1460 and to execute mapapplication 1434. The map application 1434 may include a plurality ofengines for facilitating the map functions of the mobile device 1400.For example, the map application 1434 may include a physical positionengine 1435, a three-dimensional movement mode engine 1436, an originposition determination engine 1437, a distance measurement engine 1439,a virtual position determination engine 1440, an image rendering engine1441, an image display engine 1442, and a cooperative map interfaceengine 1443.

Although shown and described as having a certain number of separateengines, it is contemplated that the map application 1434 may include agreater or fewer number of engines, and/or that the functions ofmultiple engines described herein may be combined.

The physical position engine 1435 may, in conjunction with the processor1462, request a current physical position and orientation of the mobiledevice 1430 from the GPS 1466 and/or one or more sensors 1472 (e.g., acompass, a gyroscope, etc.) in some embodiments. In response to therequest, the GPS 1466 may generate a set of coordinates (e.g., GPScoordinates, latitude and longitude coordinates, etc.) indicative of thecurrent location of the mobile device 1400 and provide those coordinatesto the physical position engine 14235. Similarly, the sensors maygenerate orientation data of the mobile device 1400 and provide theorientation data to the physical position engine 1435 (e.g., a cardinaldirection). The physical position engine 1435 may, in conjunction withthe processor 1462 and the communication subsystem 1474, transmit thecoordinates and the orientation data to the map server 1215. The mapserver 1215 may, in conjunction with the processor 1320, query the mapdata datastore 1317 with the coordinates and the orientation data toretrieve map data corresponding to the coordinates from the perspectiveof the orientation included in the orientation data, and transmit themap data back to the mobile device 1400 via the communication subsystem1325.

In some embodiments, the physical position engine 1435 may receive inputcorresponding to a physical position via means other than the GPS 1466.For example, the physical position engine 1435 may receive an indicatorof a location (e.g., an address, coordinates, a business name, etc.) viauser interface 1464, which may include a physical or virtual keyboard insome embodiments. The physical position engine 1435 may then, inconjunction with the processor 1462 and the communication subsystem1474, transmit the indicator to the map server 1215. The map server 1215may, in conjunction with the processor 1320, query the map datadatastore 1317 with the indicator to retrieve map data corresponding tothe indicator, and transmit the map data back to the mobile device 1400via the communication subsystem 1325.

Although shown and described as being stored in the map data datastore1317, it is contemplated that in some embodiments, certain map data maybe cached and stored locally on the mobile device 1400. For example,frequently used map data (e.g., map data including a home or worklocation) may be stored in the memory 1432, and thus may not need to berequested from the map server 1215. This may reduce data consumption bythe mobile device 1400 and decrease processing time needed to displayfrequently used map data.

A. Registration

Once the map data has been retrieved (either locally or from the mapserver 1215), a map view displaying the map data may be displayed on thedisplay 1468. The map view may include a set of map objectscorresponding to a location within a map from an initial virtualposition of a virtual camera. The virtual camera may have an initialvirtual position of being overhead of the entered address. Exemplary mapobjects may include features (e.g., roads, buildings, monuments, parks,landmarks, etc.) at or around the entered location.

From the map view, the user may interact with the user interface 1464 toselect a three-dimensional movement mode that uses the physical camera1470. For example, a user may select a button on the user interfacecorresponding to the three-dimensional movement mode. In anotherexample, the three-dimensional movement mode may be automaticallyactivated upon a user's selection of a three-dimensional map orsatellite view. Selection of the three-dimensional mode may activate thethree-dimensional movement mode engine 1436. The three-dimensionalmovement mode engine 1436 may, in conjunction with the processor 1462,activate the physical camera 1470. The three-dimensional movement modeengine 1436 may capture one or more first images of a physicalenvironment within which the mobile device 1430 resides using thephysical camera 1470. In other words, once activated, the physicalcamera 1470 may capture an image of the environment positioned in frontof and around the physical camera 1470. This image may be provided tothe origin position determination engine 1437.

The origin position determination engine 1437 may, in conjunction withthe processor 1462, determine an origin (i.e., initial) position of thephysical camera in the physical environment using the one or more firstimages. For example, the origin position determination engine 1437 mayanalyze the images captured by the physical camera 1470 to determinethat the physical camera 1470 is positioned directly in front of a treewith a blue house to the left of the tree and a red house to the rightof the tree. The origin position determination engine 1437 may registerthe physical positions of these objects. In some embodiments, the originposition corresponds to a physical position of the physical camera 1470when the user input is received activating the three-dimensionalmovement mode. In some embodiments, the origin position may defaultautomatically at the center of the map. This origin position may be usedto thereafter determine physical movement of the mobile device afteractivation of the three-dimensional movement mode, as described furtherherein.

The origin position determination engine 1437 may further, inconjunction with the processor 1462, specify a set of physical positionsof a set of three-dimensional objects of the map relative to the originposition. The set of three-dimensional objects may correspond to the setof map objects. For example, the building indicated by the pin at “2Embarcadero Center” of FIG. 3 may be a map object corresponding to thethree-dimensional building indicated by the pin at “2 EmbarcaderoCenter” in FIG. 3. The physical position of the three-dimensionalbuilding indicated by the pin in FIG. 3 relative to the origin positionmay be straight ahead (e.g., corresponding to the tree in the aboveexample). The physical position of the building to the left of the pinrelative to the origin position may be ahead and to the left (e.g.,corresponding to the blue house in the above example). The physicalposition of the building to the right of the pin relative to the originposition may be head and to the right (e.g., corresponding to the redhouse in the above example).

In some embodiments, the set of physical positions of the set ofthree-dimensional objects are specified at default positions and/ororientations with respect to the mobile device 1430. For example, thedefault positions may be dependent on the tilt angle of the mobiledevice 1430. In some embodiments, if the physical camera 1470 is at anangle greater than 45 degrees from face down (e.g., face up), thedefault positions may be at a lesser default tilt angle (e.g., 15degrees). This may be beneficial in that it does not allow impracticalor non-useful views of objects to be rendered (e.g., views of the sky).Similarly, the default positions may include default heights and/ordistances from the ground or other map objects. In some embodiments, theset of physical positions of the set of three-dimensional objects arespecified based on angular coordinates of the mobile device 1430. Thesephysical positions may be used to render views of the three-dimensionalobjects, as described further herein.

B. Distance Measurement

The distance measurement engine 1439 may, in conjunction with theprocessor 1462, detect movement of the physical camera 1470 from theorigin position. This movement may be detected using any suitablemethod. In some embodiments, this movement may be detected using one ormore sensors 1472. The sensors may include, for example, a gyroscope, anaccelerometer, a compass, and/or the like. For example, a gyroscope mayindicate angular rotation of the mobile device 1400 downward. In anotherexample, an accelerometer may indicate movement of the mobile deviceforward. In still another example, a compass may indicate movement ofthe mobile device from a north-facing position to a northeast-facingposition. In still another example, a GPS 1466 may indicate movement ofthe mobile device 1400 from an initial position to a new position. Insome embodiments, the movement detected by the sensors 1472 alone may besufficient to determine the new physical position of the mobile device1400, and thus the new virtual position of the virtual camera. Forexample, the mobile device 1400 may include an orientation sensor. Anangular change between the origin position of the physical camera 1470and the current physical position of the physical camera 1470 may bedetermined using the orientation sensor. The angular change may be usedto determine the updated virtual position, thereby rotating the virtualcamera.

In some embodiments, movement of the mobile device 1400 from the originposition may alternatively or additionally be detected by comparingsubsequently captured images from the physical camera 1470 to the imagestaken at the origin position. In these embodiments, the distancemeasurement engine 1439 may, in conjunction with the processor 1462,capture one or more second images of the physical environment withinwhich the mobile device resides using the physical camera 1470. Thedistance measurement engine 1439 may capture the second images aftermovement of the mobile device 1400.

The distance measurement engine 1439 may determine a current physicalposition of the physical camera 1470 with respect to the origin positionbased on the one or more second images. For example, the distancemeasurement engine 1439 may compare the first images to the secondimages to determine how and to what degree the physical camera 1470 hasmoved. For example, a first image may show a group of five treesstraight ahead with a beige house to the left and brown townhomes to theright. A second image may show a group of two trees straight ahead withthe beige house zoomed in to the right and a red car to the left. Thedistance measurement engine 1439 may analyze these images to determinethat the physical camera 1470 has shifted to the left and zoomed in withrespect to the origin position. The distance measurement engine 1439 mayfurther estimate the amount of movement of the physical camera 1470,e.g., 10 feet to the left. In some embodiments, the current physicalposition of the physical camera 1470 is specified as a six-dimensioncoordinate of translation coordinates and angular coordinates.

As an example, the sensor measurements and/or images taken by thephysical camera 1470 can be used to determine a movement vector relativeto the initial position of the mobile device 1400. The initial positionitself can be defined at a position relative to an origin, and thecurrent position of the mobile device 1400 can be composed of these twovectors.

The sensors can be used to determine movement. For example, it can beestimated that the mobile device 1400 has moved 30 centimeters based onan accelerometer reading. The 30 centimeters can define a sphere aroundwhich the mobile device 1400 could have moved. If the accelerometer is athree-dimensional accelerometer, the direction of movement can bedetermined, thereby providing an estimate for the relative movementvector. The images can be analyzed to refine the position. For example,the new distance (e.g., pixels on the image) between objects can beanalyzed and compared to the pixel distance for the initial image. Thepixel distance can be translated to an actual distance based on cameraparameters, e.g., zoom setting and depth of field analysis. Theaccelerometer position and the image position can be used to determinethe current movement vector, e.g., by taking an average. In otherembodiments, the image position can use the accelerometer position as aninput, so the image analysis can start with an approximate value. Insome embodiments, the image position can use a gyroscope or compassreading as input, so the image analysis can start with an approximateangular or directional orientation.

C. Map Rendering at New Position

The virtual position determination engine 1440 may, in conjunction withthe processor 1462, determine an updated virtual position andorientation of the virtual camera based on the current physical positionand orientation of the physical camera. The first and second imagestaken by the physical camera 1470 can be used to determine a movementvector relative to the initial position of the mobile device 1400. Theinitial position itself can be defined at a position relative to anorigin, and the current position of the mobile device 1400 can becomposed of these two vectors. The current position may then be used todetermine an updated virtual position. For example, the virtual positiondetermination engine 1440 may determine that the virtual camera shouldbe shifted right and pointed downward based on the movement of thephysical camera 1470 to the right and its angular movement downward. Insome embodiments, the virtual position determination engine 1440 mayscale the estimated amount of movement of physical camera 1470 to adifferent amount of movement of the virtual camera. For example, everyinch of movement of the physical camera 1470 may be scaled to 10 feet ofmovement of the virtual camera.

The image rendering engine 1441 may, in conjunction with the processor1462, render an image of the set of three-dimensional objects based onthe updated virtual position of the virtual camera. The image renderingengine 1441 may provide the rendered image to the image display engine1442. The image display engine 1442 may, in conjunction with theprocessor 1462, display the rendered image on the display 1468.

D. Map Overlay on Camera Images

In some embodiments, the image display engine 1442 may overlay therendered map image on the second images (or any subsequent imagescaptured in real time by the physical camera 1470). In some embodiments,the image display engine 1442 may, in conjunction with the processor1462, identify a surface in the real time images captured by thephysical camera 1470. For example, the image display engine 1442 mayidentify a horizontal surface (or other suitable flat surface) presentin the images, such as a table, a shelf, or a floor. The image displayengine 1442 may overlay the rendered map image (e.g., athree-dimensional view of buildings, streets and/or other features) onthe identified surface in the camera images. For example, buildings maybe rendered onto the top surface of an image of a desk such that thebase of the buildings appear to be positioned on the desk with thebuildings protruding from the desk. The mobile device 1400 may then bemoved around the desk, still pointed at the desk, and the map image maybe continuously rendered to reflect the changes in position andorientation, such that the map image appears to be a three-dimensionalmodel on the desk.

E. Cooperation Between Mobile Devices

The mobile device 1400 may be implemented by a host device and/or aclient device as described herein. For example, when the mobile device1400 is a host device, the host device may perform the functionsdescribed above with respect to FIG. 14 and may further include acooperative map interface engine 1443. The cooperative map interfaceengine 1443 may, in conjunction with the processor 1462, receive anorigin position of the host device from the origin positiondetermination engine 1437, as well as a series of virtual positions ofthe host device from the virtual position determination engine 1440. Thecooperative map interface engine 1443 may use the communicationsubsystem 1474 to transmit this data to a client device for rendering ofmap images on the client device when the client device is in a “followermode”.

In another example, the mobile device 1400 may be a client device. Forexample, when the client device is in a “discovery mode” or a combinedmode, the client device may perform the functions described above withrespect to FIG. 14 and may further include a cooperative map interfaceengine 1443. The cooperative map interface engine 1443 may, inconjunction with the processor 1462, receive an origin position of thehost device and a series of virtual positions of the host device fromthe host device. The client device may toggle between following the mapimages displayed by the host device and independently accessing othermap images, as controlled by the physical position engine 1435, thedistance measurement engine 1439, the virtual position determinationengine 1440, etc.

VI. Example Device

FIG. 15 is a block diagram of an example device 1500, which may be amobile device (e.g., a host device or client device) as describedherein. Device 1500 generally includes computer-readable medium 1502, aprocessing system 1504, an Input/Output (I/O) subsystem 1506, wirelesscircuitry 1508, and audio circuitry 1510 including speaker 1550 andmicrophone 1552. These components may be coupled by one or morecommunication buses or signal lines 1503. Device 1500 can be anyportable electronic device, including a handheld computer, a tabletcomputer, a mobile phone, laptop computer, tablet device, media player,personal digital assistant (PDA), a key fob, a car key, an access card,a multi-function device, a mobile phone, a portable gaming device, a cardisplay unit, or the like, including a combination of two or more ofthese items.

It should be apparent that the architecture shown in FIG. 15 is only oneexample of an architecture for device 1500, and that device 1500 canhave more or fewer components than shown, or a different configurationof components. The various components shown in FIG. 15 can beimplemented in hardware, software, or a combination of both hardware andsoftware, including one or more signal processing and/or applicationspecific integrated circuits.

Wireless circuitry 1508 is used to send and receive information over awireless link or network to one or more other devices' conventionalcircuitry such as an antenna system, an RF transceiver, one or moreamplifiers, a tuner, one or more oscillators, a digital signalprocessor, a CODEC chipset, memory, etc. Wireless circuitry 1508 can usevarious protocols, e.g., as described herein.

Wireless circuitry 1508 is coupled to processing system 1504 viaperipherals interface 1516. Interface 1516 can include conventionalcomponents for establishing and maintaining communication betweenperipherals and processing system 1504. Voice and data informationreceived by wireless circuitry 1508 (e.g., in speech recognition orvoice command applications) is sent to one or more processors 1518 viaperipherals interface 1516. One or more processors 1518 are configurableto process various data formats for one or more application programs1534 stored on medium 1502.

Peripherals interface 1516 couple the input and output peripherals ofthe device to processor 1518 and computer-readable medium 1502. One ormore processors 1518 communicate with computer-readable medium 1502 viaa controller 1520. Computer-readable medium 1502 can be any device ormedium that can store code and/or data for use by one or more processors1518. Medium 1502 can include a memory hierarchy, including cache, mainmemory and secondary memory.

Device 1500 also includes a power system 1542 for powering the varioushardware components. Power system 1542 can include a power managementsystem, one or more power sources (e.g., battery, alternating current(AC)), a recharging system, a power failure detection circuit, a powerconverter or inverter, a power status indicator (e.g., a light emittingdiode (LED)) and any other components typically associated with thegeneration, management and distribution of power in mobile devices.

In some embodiments, device 1500 includes a camera 1544 (e.g., aphysical camera). In some embodiments, device 1500 includes sensors1546. Sensors 1546 can include accelerometers, compasses, gyroscopes,pressure sensors, audio sensors, light sensors, barometers, and thelike. Sensors 1546 can be used to sense location aspects, such asauditory or light signatures of a location.

In some embodiments, device 1500 can include a GPS receiver, sometimesreferred to as a GPS unit 1548. A mobile device can use a satellitenavigation system, such as the Global Positioning System (GPS), toobtain position information, timing information, altitude, or othernavigation information. During operation, the GPS unit can receivesignals from GPS satellites orbiting the Earth. The GPS unit analyzesthe signals to make a transit time and distance estimation. The GPS unitcan determine the current position (current location) of the mobiledevice. Based on these estimations, the mobile device can determine alocation fix, altitude, and/or current speed. A location fix can begeographical coordinates such as latitudinal and longitudinalinformation.

One or more processors 1518 run various software components stored inmedium 1502 to perform various functions for device 1500. In someembodiments, the software components include an operating system 1522, acommunication module (or set of instructions) 1524, a location module(or set of instructions) 1526, a cooperative map interface module thatimplements a “follower mode”, a “discovery mode”, and/or a combinedmode, as described herein, and other applications (or set ofinstructions) 1534.

Operating system 1522 can be any suitable operating system, includingiOS, Mac OS, Darwin, RTXC, LINUX, UNIX, OS X, WINDOWS, or an embeddedoperating system such as VxWorks. The operating system can includevarious procedures, sets of instructions, software components and/ordrivers for controlling and managing general system tasks (e.g., memorymanagement, storage device control, power management, etc.) andfacilitates communication between various hardware and softwarecomponents.

Communication module 1524 facilitates communication with other devicesover one or more external ports 1536 or via wireless circuitry 1508 andincludes various software components for handling data received fromwireless circuitry 1508 and/or external port 1536. External port 1536(e.g., USB, FireWire, Lightning connector, 60-pin connector, etc.) isadapted for coupling directly to other devices or indirectly over anetwork (e.g., the Internet, wireless LAN, etc.).

Location/motion module 1526 can assist in determining the currentposition (e.g., coordinates or other geographic location identifiers)and motion of device 1500. Modern positioning systems includesatellite-based positioning systems, such as Global Positioning System(GPS), cellular network positioning based on “cell IDs,” and Wi-Fipositioning technology based on a Wi-Fi networks. GPS also relies on thevisibility of multiple satellites to determine a position estimate,which may not be visible (or have weak signals) indoors or in “urbancanyons.” In some embodiments, location/motion module 1526 receives datafrom GPS unit 1548 and analyzes the signals to determine the currentposition of the mobile device. In some embodiments, location/motionmodule 1526 can determine a current location using Wi-Fi or cellularlocation technology. For example, the location of the mobile device canbe estimated using knowledge of nearby cell sites and/or Wi-Fi accesspoints with knowledge also of their locations. Information identifyingthe Wi-Fi or cellular transmitter is received at wireless circuitry 1508and is passed to location/motion module 1526. In some embodiments, thelocation module receives the one or more transmitter IDs. In someembodiments, a sequence of transmitter IDs can be compared with areference database (e.g., Cell ID database, Wi-Fi reference database)that maps or correlates the transmitter IDs to position coordinates ofcorresponding transmitters, and computes estimated position coordinatesfor device 1500 based on the position coordinates of the correspondingtransmitters. Regardless of the specific location technology used,location/motion module 1526 receives information from which a locationfix can be derived, interprets that information, and returns locationinformation, such as geographic coordinates, latitude/longitude, orother location fix data.

The cooperative map interface module 1528 can activate automatically orin response to user input, such as a gesture, movement, and/or selectionon a user interface. Once activated, the cooperative map interfacemodule 1528 may capture image data using the camera 1544. Thecooperative map interface module 1528 can determine whether the mobiledevice 1500 has been moved and move a virtual camera showing a map, asdescribed further herein, and/or transmit that data to another mobiledevice 1500.

The one or more applications programs 1534 on the mobile device caninclude any applications installed on the device 1500, including withoutlimitation, a browser, address book, contact list, email, instantmessaging, word processing, keyboard emulation, widgets, JAVA-enabledapplications, encryption, digital rights management, voice recognition,voice replication, a music player (which plays back recorded musicstored in one or more files, such as MP3 or AAC files), etc.

There may be other modules or sets of instructions (not shown), such asa graphics module, a time module, etc. For example, the graphics modulecan include various conventional software components for rendering,animating and displaying graphical objects (including without limitationtext, web pages, icons, digital images, animations and the like) on adisplay surface. In another example, a timer module can be a softwaretimer. The timer module can also be implemented in hardware. The timemodule can maintain various timers for any number of events.

The I/O subsystem 1506 can be coupled to a display system (not shown),which can be a touch-sensitive display. The display system displaysvisual output to the user in a GUI. The visual output can include text,graphics, video, and any combination thereof. Some or all of the visualoutput can correspond to user-interface objects. A display can use LED(light emitting diode), LCD (liquid crystal display) technology, or LPD(light emitting polymer display) technology, although other displaytechnologies can be used in other embodiments.

In some embodiments, I/O subsystem 1506 can include a display and userinput devices such as a keyboard, mouse, and/or track pad. In someembodiments, I/O subsystem 1506 can include a touch-sensitive display. Atouch-sensitive display can also accept input from the user based onhaptic and/or tactile contact. In some embodiments, a touch-sensitivedisplay forms a touch-sensitive surface that accepts user input. Thetouch-sensitive display/surface (along with any associated modulesand/or sets of instructions in medium 1502) detects contact (and anymovement or release of the contact) on the touch-sensitive display andconverts the detected contact into interaction with user-interfaceobjects, such as one or more soft keys, that are displayed on the touchscreen when the contact occurs. In some embodiments, a point of contactbetween the touch-sensitive display and the user corresponds to one ormore digits of the user. The user can make contact with thetouch-sensitive display using any suitable object or appendage, such asa stylus, pen, finger, and so forth. A touch-sensitive display surfacecan detect contact and any movement or release thereof using anysuitable touch sensitivity technologies, including capacitive,resistive, infrared, and surface acoustic wave technologies, as well asother proximity sensor arrays or other elements for determining one ormore points of contact with the touch-sensitive display.

Further, the I/O subsystem 1506 can be coupled to one or more otherphysical control devices (not shown), such as pushbuttons, keys,switches, rocker buttons, dials, slider switches, sticks, LEDs, etc.,for controlling or performing various functions, such as power control,speaker volume control, ring tone loudness, keyboard input, scrolling,hold, menu, screen lock, clearing and ending communications and thelike. In some embodiments, in addition to the touch screen, device 1500can include a touchpad (not shown) for activating or deactivatingparticular functions. In some embodiments, the touchpad is atouch-sensitive area of the device that, unlike the touch screen, doesnot display visual output. The touchpad can be a touch-sensitive surfacethat is separate from the touch-sensitive display or an extension of thetouch-sensitive surface formed by the touch-sensitive display.

In some embodiments, some or all of the operations described herein canbe performed using an application executing on the user's device.Circuits, logic modules, processors, and/or other components may beconfigured to perform various operations described herein. Those skilledin the art will appreciate that, depending on implementation, suchconfiguration can be accomplished through design, setup,interconnection, and/or programming of the particular components andthat, again depending on implementation, a configured component might ormight not be reconfigurable for a different operation. For example, aprogrammable processor can be configured by providing suitableexecutable code; a dedicated logic circuit can be configured by suitablyconnecting logic gates and other circuit elements; and so on.

Any of the software components or functions described in thisapplication may be implemented as software code to be executed by aprocessor using any suitable computer language such as, for example,Java, C, C++, C#, Objective-C, Swift, or scripting language such as Perlor Python using, for example, conventional or object-orientedtechniques. The software code may be stored as a series of instructionsor commands on a computer readable medium for storage and/ortransmission. A suitable non-transitory computer readable medium caninclude random access memory (RAM), a read only memory (ROM), a magneticmedium such as a hard-drive or a floppy disk, or an optical medium, suchas a compact disk (CD) or DVD (digital versatile disk), flash memory,and the like. The computer readable medium may be any combination ofsuch storage or transmission devices.

Computer programs incorporating various features of the presentdisclosure may be encoded on various computer readable storage media;suitable media include magnetic disk or tape, optical storage media,such as compact disk (CD) or DVD (digital versatile disk), flash memory,and the like. Computer readable storage media encoded with the programcode may be packaged with a compatible device or provided separatelyfrom other devices. In addition, program code may be encoded andtransmitted via wired optical, and/or wireless networks conforming to avariety of protocols, including the Internet, thereby allowingdistribution, e.g., via Internet download. Any such computer readablemedium may reside on or within a single computer product (e.g., asolid-state drive, a hard drive, a CD, or an entire computer system),and may be present on or within different computer products within asystem or network. A computer system may include a monitor, printer, orother suitable display for providing any of the results mentioned hereinto a user.

A recitation of “a”, “an” or “the” is intended to mean “one or more”unless specifically indicated to the contrary. The use of “or” isintended to mean an “inclusive or,” and not an “exclusive or” unlessspecifically indicated to the contrary. Reference to a “first” elementdoes not necessarily require that a second element be provided.Moreover, reference to a “first” or a “second” element does not limitthe referenced element to a particular location unless expressly stated.

All patents, patent applications, publications, and descriptionsmentioned herein are incorporated by reference in their entirety for allpurposes. None is admitted to be prior art.

The foregoing description, for purposes of explanation, used specificnomenclature to provide a thorough understanding of the describedembodiments. However, it will be apparent to one skilled in the art thatthe specific details are not required in order to practice the describedembodiments. Thus, the foregoing descriptions of the specificembodiments described herein are presented for purposes of illustrationand description. They are not taught to be exhaustive or to limit theembodiments to the precise forms disclosed. Many modifications andvariations are possible in view of the above teachings.

What is claimed is:
 1. A method of providing a three-dimensional (3D)virtual map on a display of a client mobile device, the methodcomprising, performing, by the client mobile device: retrieving, from amap server, a set of map objects corresponding to a region of the 3Dvirtual map; receiving, over a network, an origin position in the regionof the 3D virtual map, the origin position being specified by an initialposition of a host mobile device, the initial position determined by:receiving one or more first images of a physical environment withinwhich the host mobile device resides, using a camera on the host mobiledevice; and determining the initial position of the host mobile devicein the physical environment using the one or more first images;receiving, over the network, a stream of virtual positions of the hostmobile device in the 3D virtual map corresponding to physical movementsof the host mobile device, wherein the stream of virtual positions aredetermined by one of more second images of the physical environmentusing the camera from the host mobile device; and rendering anddisplaying a series of images of the 3D virtual map on the client mobiledevice using the stream of virtual positions of the host mobile deviceand the set of map objects corresponding to the region of the 3D virtualmap.
 2. The method of claim 1, further comprising: receiving ageographic indicator from the host mobile device, wherein the clientmobile device retrieves the set of map objects corresponding to theregion of the 3D virtual map based on the geographic indicator.
 3. Themethod of claim 2, wherein the geographic indicator includes the originposition.
 4. The method of claim 1, further comprising: rendering anddisplaying a trace of the stream of virtual positions of the host mobiledevice on the 3D virtual map.
 5. A method of providing athree-dimensional (3D) virtual map on a display of a client mobiledevice, the method comprising, performing, by a host mobile devicehaving a physical camera: retrieving, from a map server, a set of mapobjects corresponding to a region of the 3D virtual map; capturing oneor more first images of a physical environment within which the hostmobile device resides, using the physical camera; determining an initialposition of the host mobile device in the physical environment using theone or more first images; specifying an origin position in the region ofthe 3D virtual map based on the initial position; generating a stream ofvirtual positions of the host mobile device in the 3D virtual mapcorresponding to physical movements of the host mobile device, whereinthe stream of virtual positions are determined by one of more secondimages of the physical environment using the physical camera from thehost mobile device; and transmitting the origin position, the initialposition, and the stream of virtual positions to the client mobiledevice, wherein the client mobile device renders and displays a seriesof images of the 3D virtual map on the client mobile device using theorigin positions, the initial positions, the stream of virtual positionsof the host mobile device, and the set of map objects respectivelydownloaded from the map server by the client mobile device.
 6. Themethod of claim 5, further comprising: rendering and displaying theseries of images of the 3D virtual map on the host mobile device usingthe stream of virtual positions of the host mobile device.
 7. The methodof claim 6, further comprising: rendering and displaying a trace of thestream of virtual positions of the host mobile device on the 3D virtualmap.
 8. The method of claim 5, wherein the set of map objectscorresponding to the region of the 3D virtual map are retrieved using ageographic indicator specified by the host mobile device.
 9. The methodof claim 8, wherein the geographic indicator includes the originposition.
 10. The method of claim 5, wherein the origin position and thestream of virtual positions are transmitted to the client mobile deviceover a network.
 11. The method of claim 5, wherein the set of mapobjects are retrieved from the map server over a first network, whereinthe origin position and the stream of virtual positions are transmittedto the client mobile device over a second network, and wherein the firstnetwork and the second network are different.
 12. A mobile client devicecomprising: a display; a physical camera communicably coupled with thedisplay; a processor; and a memory coupled to the processor, the memorystoring instructions, which when executed by the processor, cause themobile client device to perform operations including: retrieving, from amap server, a set of map objects corresponding to a region of athree-dimensional (3D) virtual map; receiving, over a network, an originposition in the region of the 3D virtual map, the origin position beingspecified by an initial position of a host mobile device, the initialposition determined by: receiving one or more first images of a physicalenvironment within which the host mobile device resides, using a cameraon the host mobile device; and determining the initial position of thehost mobile device in the physical environment using the one or morefirst images; receiving, over the network, a stream of virtual positionsof the host mobile device in the 3D virtual map corresponding tophysical movements of the host mobile device, wherein the stream ofvirtual positions are determined by one of more second images of thephysical environment using the camera from the host mobile device; andrendering and displaying a series of images of the 3D virtual map on theclient mobile device using the stream of virtual positions of the hostmobile device and the set of map objects corresponding to the region ofthe 3D virtual map.
 13. The mobile client device of claim 12, whereinthe operations further comprise: receiving a geographic indicator fromthe host mobile device, wherein the client mobile device retrieves theset of map objects corresponding to the region of the 3D virtual mapbased on the geographic indicator.
 14. The mobile client device of claim13, wherein the geographic indicator includes the origin position. 15.The mobile client device of claim 12, wherein the operations furthercomprise: rendering and displaying a trace of the stream of virtualpositions of the host mobile device on the 3D virtual map.
 16. Anon-transitory computer readable medium storing a plurality ofinstructions that, when executed by one or more processors of a computerdevice, cause the one or more processors to perform operations forproviding a view of a three-dimensional (3D) virtual map on a display ofa client mobile device having a physical camera communicably coupledwith the display, the operations comprising: retrieving, from a mapserver, a set of map objects corresponding to a region of thethree-dimensional (3D) virtual map; receiving, over a network, an originposition in the region of the 3D virtual map, the origin position beingspecified by an initial position of a host mobile device, the initialposition determined by: receiving one or more first images of a physicalenvironment within which the host mobile device resides, using a cameraon the host mobile device; and determining the initial position of thehost mobile device in the physical environment using the one or morefirst images; receiving, over the network, a stream of virtual positionsof the host mobile device in the 3D virtual map corresponding tophysical movements of the host mobile device, wherein the stream ofvirtual positions are determined by one of more second images of thephysical environment using the camera from the host mobile device; andrendering and displaying a series of images of the 3D virtual map on theclient mobile device using the stream of virtual positions of the hostmobile device and the set of map objects corresponding to the region ofthe 3D virtual map.
 17. The non-transitory computer readable medium ofclaim 16, the operations further comprise: receiving a geographicindicator from the host mobile device, wherein the client mobile deviceretrieves the set of map objects corresponding to the region of the 3Dvirtual map based on the geographic indicator.
 18. The non-transitorycomputer readable medium of claim 17, wherein the geographic indicatorincludes the origin position.
 19. The non-transitory computer readablemedium of claim 16, wherein the operations further comprise: renderingand displaying a trace of the stream of virtual positions of the hostmobile device on the 3D virtual map.
 20. The non-transitory computerreadable medium of claim 16, the operations further comprise retrieving,from a map server, a set of map objects corresponding to a region of the3D virtual map.